1. Related Cases
METHOD, SYSTEM, AND PROGRAM FOR MANAGING MEMORY FOR DATA TRANSMISSION THROUGH A NETWORK, Ser. No. 10/683,941, filed Oct. 9, 2003; METHOD, SYSTEM, AND PROGRAM FOR MANAGING VIRTUAL MEMORY, Ser. No. 10/747,920, filed Dec. 29, 2003; METHOD, SYSTEM, AND PROGRAM FOR CACHING A VIRTUALIZED DATA STRUCTURE TABLE, Ser. No. 10/882,557, filed Jun. 30, 2004; and MESSAGE CONTEXT BASED TCP TRANSMISSION, Ser. No. 10/809,077, filed Mar. 24, 2004.
2. Description of Related Art
In a network environment, a network adapter on a host computer, such as an Ethernet controller, Fibre Channel controller, etc., will receive Input/Output (I/O) requests or responses to I/O requests initiated from the host computer.
Often, the host computer operating system includes a device driver to communicate with the network adapter hardware to manage I/O requests to transmit over a network. The host computer may also employ a protocol which packages data to be transmitted over the network into packets, each of which contains a destination address as well as a portion of the data to be transmitted. Data packets received at the network adapter are often stored in a packet buffer. A transport protocol layer can process the packets received by the network adapter that are stored in the packet buffer, and access any I/O commands or data embedded in the packet.
For instance, the computer may employ the TCP/IP (Transmission Control Protocol Internet Protocol) (to encode and address data for transmission, and to decode and access the payload data in the TCP/IP packets received at the network adapter. IP specifies the format of packets, also called datagrams, and the addressing scheme. TCP is a higher level protocol which establishes a connection between a destination and a source and provides a byte-stream, reliable, full-duplex transport service. Another protocol, Remote Direct Memory Access (RDMA) on top of TCP provides, among other operations, direct placement of data at a specified memory location at the destination.
A device driver, application or operating system can utilize significant host processor resources to handle network transmission requests to the network adapter. One technique to reduce the load on the host processor is the use of a TCP/IP Offload Engine (TOE) in which TCP/IP protocol related operations are embodied in the network adapter hardware as opposed to the device driver or other host software, thereby saving the host processor from having to perform some or all of the TCP/IP protocol related operations. Similarly, an RDMA-enabled NIC (RNIC) offloads RDMA and transport related operations from the host processor(s).
Offload engines and other devices frequently utilize memory, often referred to as a buffer, to store or process data. Buffers have been embodied using physical memory which stores data, usually on a short term basis, in integrated circuits, an example of which is a random access memory or RAM. Typically, data can be accessed relatively quickly from such physical memories. Also, a buffer is usually arranged as a set of physically contiguous memory locations, that is, memory locations having contiguous physical addresses. A host computer often has additional physical memory such as hard disks and optical disks to store data on a longer term basis. These nonintegrated circuit based physical memories tend to retrieve data more slowly than the integrated circuit physical memories.
The operating system of a computer typically utilizes a virtual memory space which is often much larger than the memory space of the physical memory of the computer. FIG. 1 shows an example of a virtual memory space 50 and a short term physical memory space 52. The memory space of a long term physical memory such as a hard drive is indicated at 54. The data to be sent in a data stream or the data received from a data stream may initially be stored in noncontiguous portions, that is, nonsequential memory addresses, of the various memory devices. For example, two portions indicated at 10a and 10b may be stored in the physical memory in noncontiguous portions of the short term physical memory space 52 while another portion indicated at 10c may be stored in a long term physical memory space provided by a hard drive as shown in FIG. 1. The operating system of the computer uses the virtual memory address space 50 to keep track of the actual locations of the portions 10a, 10b and 10c of the datastream 10. Thus, a portion 50a of the virtual memory address space 50 is mapped to the actual physical memory addresses of the physical memory space 52 in which the data portion 10a is stored. In a similar fashion, a portion 50b of the virtual memory address space 50 is mapped to the actual physical memory addresses of the physical memory space 52 in which the data portion 10b is stored. The datastream 10 frequently resides in contiguous virtual memory address space while mapped into noncontiguous physical memory space). Furthermore, a portion 50c of the virtual memory address space 50 is mapped to the physical memory addresses of the long term hard drive memory space 54 in which the data portion 10c is stored. A blank portion 50d represents an unassigned or unmapped portion of the virtual memory address space 50.
FIG. 2 shows an example of a typical system translation and protection table (TPT) 60 which the operating system utilizes to map virtual memory addresses to real physical memory addresses with protection at the process level. Thus, the virtual memory address of the virtual memory space 50a may start at virtual memory address 0X1000, for example, which is mapped to a physical memory address 8AEF000, for example of the physical memory space 52. The system TPT table 60 does not have any physical memory addresses which correspond to the virtual memory addresses of the virtual memory address space 50d because the virtual memory space 50d has not yet been mapped to physical memory space.
In known systems, portions of the virtual memory space 50 may be assigned to a device or software module for use by that module so as to provide memory space for buffers. Also, in some known designs, an Input/Output (I/O) device such as a network adapter or a storage controller may have the capability of directly placing data into an application buffer or other, memory area. Such a direct data placement capability can reduce or eliminate intermediate buffering in which data is placed in a temporary buffer prior to being transferred to the intended memory location of the system. A Remote Direct Memory Access (RDMA) enabled Network Interface Card (RNIC) is an example of an I/O device which can perform direct data placement. An RNIC can support defined operations (also referred to as “semantics”) including RDMA Write, RDMA Read and Send/Receive, for memory to memory data transfers across a network.
The address of the application buffer which is the destination of the RDMA operation is frequently carried in the RDMA packets in some form of a buffer identifier and a virtual address or offset. The buffer identifier identifies which buffer the data is to be written to or read from. The virtual address or offset carried by the packets identifies the location within the identified buffer for the specified direct memory operation.
In order to perform direct data placement, an I/O device typically maintains its own translation and protection table (TPT), an example of which is shown at 70 in FIG. 3. The device TPT 70 contains data structures 72a, 72b, 72c . . . 72n, each of which is used to control access to a particular buffer as identified by an associated buffer identifier of the buffer identifiers 74a, 74b, 74c . . . 74n. The device TPT 70 further contains data structures 76a, 76b, 76c . . . 76n, each of which is used to translate the buffer identifier and virtual address or offset into physical memory addresses of the particular buffer identified by the associated buffer identifier 74a, 74b, 74c . . . 74n. Thus, for example, the data structure 76a of the TPT 70 is used by the I/O device to perform address translation for the buffer identified by the identifier 74a. Similarly, the data structure 72a is used by the I/O device to perform protection checks for the buffer identified by the buffer identifier 74a. The address translation and protection checks may be performed prior to direct data placement of the payload contained in a packet received from the network or prior to sending the data out on the network.
In order to facilitate high-speed data transfer, a device TPT such as the TPT 70 is typically managed by the I/O device and the driver software for the device. A device TPT can occupy a relatively large amount of memory. As a consequence, a device TPT is frequently resident in system memory. The I/O device may maintain a cache of a portion of the device TPT to reduce access delays.
Notwithstanding, there is a continued need in the art to improve the performance of memory usage in data transmission and other operations.